FAQ: Why is the readback accuracy different than the meter accuracy?

The difference between the readback accuracy and the meter accuracy is that the readback is accurate up to the resolution of the analog-to-digital converter (ADC), but the display can only show a certain amount of digits. So, in rounding the readback value to a value that can fit in the display, there is a certain rounding error – and that error defines the meter accuracy.

When returning the readback value over the remote interface, the actual value of the ADC is returned without any rounding applied.

Applies to: All models

Select Home to return to the table of contents for more FAQs. Visit our Power Products Forum for additional topics and discussions. Read technical postings on our Power Blog – Watt’s Up?

6030A, 6031A, 6032A, 6033A, 6035A, 6038A, 6060B, 6063B, 6621A, 6622A, 6623A, 6624A, 6625A, 6626A, 6627A, 6628A, 6629A, 6641A, 6642A, 6643A, 6644A, 6645A, 6651A, 6652A, 6653A, 6654A, 6655A, 6671A, 6672A, 6673A, 6674A, 6675A, 6680A, 6681A, 6682A, 6683A, 6684A, 6690A, 6691A, 6692A, E4356A, 6611C, 6612C, 6613C, 6614C, 6631B, 6632B, 6633B, 6634B, 66309B, 66311B, 66312A, 66319B, 66319D, 66321B, 66321D, 66332A, 66000A, 66001A, 66002A, 66101A, 66102A, 66103A, 66104A, 66105A, 66106A, 6811B, 6812B, 6813B, AC6801A, AC6802A, AC6803A, AC6804A, E4360A, E4361A, E4362A, E4366A, E4367A, E4368A, E3620A, E3630A, E3631A, E3632A, E3633A, E3634A, E3640A, E3641A, E3642A, E3643A, E3644A, E3645A, E3646A, E3647A, E3648A, E3649A, E36102A, E36103A, E36104A, E36105A, E36106A, N3300A, N3301A, N3302A, N3303A, N3304A, N3305A, N3306A, N3307A, N5740A, N5741A, N5742A, N5743A, N5744A, N5745A, N5746A, N5747A, N5748A, N5749A, N5750A, N5751A, N5752A, N5761A, N5762A, N5763A, N5764A, N5765A, N5766A, N5767A, N5768A, N5769A, N5770A, N5771A, N5772A, N6700B, N6701A, N6702A, N6705A, N6705B, N6710B, N6711A, N6712A, N6715A, N6715B, N6731B, N6732B, N6733B, N6734B, N6735B, N6736B, N6741B, N6742B, N6743B, N6744B, N6745B, N6746B, N6773A, N6774A, N6775A, N6776A, N6777A, N6751A, N6752A, N6753A, N6754A, N6755A, N6756A, N6761A, N6762A, N6763A, N6764A, N6765A, N6766A, N6781A, N6782A, N6785A, N6786A, N6783A-MFG, N6783A-BDA, N6784A, N6950A, N6951A, N6952A, N6953A, N6954A, N6970A, N6971A, N6972A, N6973A, N6974A,N6976A, N6977A, N7950A, N7951A, N7952A, N7953A, N7954A, N7970A, N7971A, N7972A, N7973A, N7974A, N7976A, N7977A, N8731A, N8732A, N8733A, N8734A, N8735A, N8736A, N8737A, N8738A, N8739A, N8740A, N8741A, N8742A, N8754A, N8755A, N8756A, N8757A, N8758A, N8759A, N8760A, N8761A, N8762A, N8920A, N8921A, N8923A, N8924A, N8925A, N8926A, N8928A, N8929A, N8930A, N8931A, N8932A, N8934A, N8935A, N8937A, N8937APV, N8940A, N8941A, N8943A, N8944A, N8945A, N8946A, N8948A, N8949A, N8950A, N8951A, N8952A, N8954A, N8955A, N8957A, N8957APV, PA2201A, U8001A, U8002A, U8031A, U8032A